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The National Coordinating Centre for Public Engagement (NCCPE) was established in 2008 to 
support culture change in relation to public engagement. In this short response, we draw on this 
experience to suggest how the PCE component of REF 2028 might be developed 


There has been significant debate about whether it is possible to assess research culture robustly. We 
believe that it is, based on our practical experience supporting culture change in the HE system and by 
learning from good practice in other sectors. We outline an approach which we believe could command 
confidence and provide a pragmatic way of implementing the ambition of REF 2028 to ‘appropriately 
recognise and reward HEIs that create conditions in which excellent research and impact can be produced.’ 


What is the problem we are trying to solve? 

e There are many problems with how universities conduct research which are significantly impacting 
on both the lived experience of people involved in research and on the quality of the work that is 
realised as a result (for instance, by limiting the diversity of people involved in research careers). 

e By weighting ‘research culture’ more heavily in the next REF, and evolving the guidance, we can 
both recognise its contribution to excellent research and engagement, & accelerate improvements. 


Why are we stuck? 
e While there seems to be broad consensus that culture is important, there has been significant push 
back on the proposed changes to the REF, focusing on the difficulties of robustly assessing it. 


How can we move forward? 

e We need to recognize that the challenge of assessing research culture is a different order challenge 
to assessing outputs and impact, for which we now have relatively robust systems in place 
developed & refined over subsequent research assessment exercises, which command confidence. 

e In particular, we need to step outside the ‘walled garden’ of research assessment. Starting with the 
questions ‘what is research culture’ and ‘what are robust indicators of a healthy research culture’ is 
focusing attention on symptoms, not causes. 

e The core challenge we need to address is how to develop and implement a robust approach to the 
development and improvement of organizational culture. If we start here, we can draw on a host of 
well-developed approaches from other sectors (for instance healthcare). We can then integrate what 
we know about research culture to these frameworks. Below, we outline two practical ways this 
might be achieved, based on the NCCPE’s extensive experience supporting culture change. 


Supporting organizational development through maturity matrices 
e The NCCPE was established in 2008 to support ‘culture change’ in regard to one specific dimension 
of research culture: how universities support engagement with the public. 


In developing our approach, we consulted widely across the sector to understand the drivers and 
determinants of culture in relation to PE. We also looked outside the HE sector to explore tried and 
tested methods to supporting organizational development. We found a very significant body of work 
to define the determinants of good organizational culture, which we triangulated with what we were 
learning within HE. We also identified a very practical approach to implementing organizational 
change in the form of Maturity Models. 

As a result, we developed a maturity matrix for public engagement. Our EDGE tool identifies nine 
dimensions of a good culture — ranging from leadership to training and development, informed by 
wider organizational theory but ‘socialized’ within HE. 

The EDGE tool has proved very useful in identifying what matters and what different levels of 
maturity look like (for instance in how PE is rewarded and recognized). It has provided a valuable 
resource both for self-improvement (many universities use it to reflect on their current support and to 
identify priority actions) and for external benchmarking: we use the EDGE tool to structure our 
assessment of HEls, though our Engage Watermark. The tool allows for a diversity of approaches to 
be valued, helps collate perspectives from inside and outside HEls, and avoids a ‘tick box’ or 
mechanistic approach. 

We would recommend that this approach is applied to the development of the PCE guidance, to 
provide a dynamic and flexible framework to define collectively the determinants of research culture 
and what increasingly mature approaches look like. We already have a robust understanding of 
what these determinants are which could be drawn upon (e.g. the work by Wellcome; the RAND 
review of high performing research departments; and the as yet unpublished work being undertaken 
by Vitae and others to develop a ‘good practice’ framework). 

Putting such a shared framework in place would be ‘step one’ in implementing the PCE guidance. It 
would provide a dynamic framework to underpin the assessment — and would be considerably 
simpler and easier to digest and apply than long lists of stand-alone factors. It would help to provide 
the kind of simplification and clarity which was noted as an aspiration in the Initial Decisions 
publication. It would also recognise the different ways universities will seek to improve and enhance 
their research culture, valuing that diversity, whilst having a robust framework to underpin it. 


Developing indicators to scaffold quality improvement 


A maturity matrix alone is not enough. We also need a robust approach to developing indicators. 
Some good work has been done already to describe what these indicators might be, building on 
what was already in place in REF 2021. 

However, there remains significant nervousness about how robust these indicators are. In order to 
develop a more robust and confident approach we need to look outside HE to how other sectors 
have gone about developing mechanisms to quality assure their processes and cultures, and to 
draw on this to scaffold our approach in HE. 

A particularly valuable place to look would be healthcare, where there has been a long recognition 
of the complex interplay of culture and service provision, and its impact on patient outcomes. A 
particularly helpful and influential model in healthcare is the one first outlined by Donalbein (1988) 
who described the value of a three-component approach — structure, process and outcomes — with 
each component having an effect on the next. Structure refers to provider attributes (e.g., 
nurse/patient ratio), process refers to the care given to the patient (e.g., patients with acute 
myocardial infarction being prescribed aspirin on discharge), and outcome is what happens to the 
patient (e.g., recovery following cardiac surgery). 

We highlight this approach to make the point that there is very valuable learning from outside the 
world of research culture which we should be drawing on. To date, sector commentary on the 
proposed approach has failed to identify the potential to integrate this kind of learning, and we risk 
‘making it up as we go along’. 


Putting it all together 


It will clearly take time to evolve an approach that commands confidence. The insights outlined 
above would provide a robust starting point for a developmental process involving the sector and 
the REF team, drawing in expertise and help from outside higher education. 

Focal points for the work could be organized as follows (adapting the insights from quality 
improvement in health care): 


Structure Developing an agreed list of the structural factors that determine research culture, building 
dimensions | on the existing literature (about research culture but also drawing on wider organisational 


development evidence). These would constitute the y axis of the maturity matrix. 


Process These would help determine the x axis of the maturity matrix. We would recommend that 
dimensions | the existing terms ‘vitality’ and ‘sustainability’ be reviewed as they are not necessarily the 
most helpful. The literature would suggest that other important ‘process’ dimensions 
include ‘inclusivity’ and ‘engagement’, for example. 

Outcome Noting that the development of Outcome indicators is challenging, we are confident that a 
indicators number of useful data points could be developed to integrate with the other dimensions. 


What next? 

It is encouraging that the REF team have invited further consultation on this challenging area, and will be 
investing in a year-long programme to develop meaningful indicators. We would recommend that this work 
also explores learning from other sectors about how to robustly approach organisational change and quality 


improvement. Ideally, a pilot process would also be initiated, to allow the wider approach to be properly 
‘road tested’. 


Given the need to do this well, extending the timeline for the next REF may also be sensible, providing the 
necessary space to develop a rigorous approach that inspires confidence in the sector, and time for the 
sector to implement this appropriately. 
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